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from among a plurality ol extracted candidates. A fibre optic endoscope with a cun^able end section may be used. 
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METHODS OF DETBCTTWG gNDQSCQPE IWSERTIQH DTRBCTIOy; 

This invention relates to methods of detecting the 
requisite direction of Insertion of an endoscope, and more 
particularly to an endoscope insertion direction detecting method 
adapted to automatically aid the Insertion of an endoscope in the 
large Intestine^ to facilitate medical inspection. 

Recently, there has been extensive use of an endoscope 
vhereby organs within a body cavity can be observed by Inserting an 
elongate insertion sheath into the body cavity and various curing 
treatments can be made by using treatment tools, inserted through 
the sheath in a treating tool channel, as required » 

In a conventional endoscope inspection, the doctor judges 
the required direction for advance of the endoscope insertion 
sheath by observing the endoscope image, n^ile Inserting the 
endoscope. 

Hovever, high levels of technique and skill are required 
to insert an endoscope in Inspecting the large Intestine. 

One object of the present Invention Is to provide an 
endoscope Insertion direction detecting method vhereby the 
endoscope insertion direction can be singly and quickly detected, 
trtillst the adverse influence of noise can be reduced, enabling the 
appropriate endoscope insertion direction to be detected more 
accurately* 



In accordance vlth the present invention a method of 
detecting a required endoscope Insertion direction includes 
extracting the form of a fold existing on the inside wall of an 
observed part from an endoscope picture image and to judge the 
endoscope insertion direction on the basis of the form of the fold 
extracted by this step. It comprises, for example, a step of 
extracting discontinuous points at vhich the brightness or colour 
in an endoscope picture image varies. 

Advantageously, after extracting said discontinuous 
points, the invention comprises the further steps of extracting 
contour segments based on said discontinuous points from respective 
divided picture images obtained by dividing the picture image 
obtained by the above-mentioned step of extracting the 
discontinuous points into a plurality of picture Images, a step of 
connecting the contour segments obtained by the above-mentioned 
step of extracting the contour segments, and a step of considering 
the line connected by the above-mentioned connecting step to be the 
form of a fold existing on the inside lall of the observed part and 
Judging the endoscope insertion direction on the basis of this form 
of the fold. The above-mentioned step of extracting the 
discontinuous points determines the degree to vhich the brightness 
of the picture image varies. The above-mentioned step of 
extracting the contour segments has, for example, a step of 
extracting contour segront candidates and a further step of 
extracting the optimum contour segment from among those 
candidates. For example, the above-mentioned step of extracting 
the candidates includes using a midified Bough transformation in 
extracting the contour segnents. 
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The above'-mentioned step o£ extracting the optimiim contour segment 
includes making a perceptual grouping to. extract the optimum 
contour segment. The above-mentioned connecting step includes, for 
example, extracting a search starting contour segment by utilising 
a pyramid quad tree structure. 

The invention will now be described with reference to the 
drawings, in vhich:- 

Figore 1 Is a flow chart shoving the steps of one 
exemplary method in accordance with the .present invention; 

Figure 2 is an explanatory view representing the 
insertion sheath of an endoscope entering into the large intestine; 

Figure 3 is a perspective view shoving the tip of the 
endoscope insertion sheath; 

Figure 4 is an explanatory view shoving the insertion 
sheath of an endoscope advancing into a bent part of the large 
intestine; 

Figure 5 is an explanatory view shoving the endoscope 
image obtained when in the state shown in Figure 4; 

Figure 6 is an explanatory view showing the insertion 
sheath of an endoscope entering into a straight part of the large 
intestine; 

Figure 7 is an explanatory view showing the endoscope 
image obtained when in the state shown in Figure S; 

Figure B is an explanatory view showing an example of a 
suitable endoscope apparatus, using a fibre-scope and externally 
fitted television camera; 



Figure 9 is an explanatory view shoving an exan^le of a 
suitable endoscope apparatus using a video-scope; 

Figure 10 is a view for explaining the use of a spatial 
filtering in the first step; 

Figure 11(a) is an explanatory viev shoving an overlapped 
SBtrix detersdning a gradient in an x-direction, and 

Figure ll(b] is an explanatory viev shoving an overlapped 
mtrix determining a gradient in a y-direction; 

Figure 12 is an explanatory viev showing an exemplary 
picture image, as obtained in the first step, vhen divided into 
small squares of 8 x 8 pixels; 

Figure 13 is an explanatory viev shoving a small square 
of 8 X 8 pixels; 

Figure 14 is an explanatory view shoving a contour 

segment 

of a fold located in a small region of 8 x 8 pixels; 

Figure 15 is an explanatory viev shoving a small sqoare 
of 8 X 8 pixels with addresses applied around the outer periphery 
to make a modified Hough transformation; 

Figure 15 is an explanatory viev shoving arranged 
elements obtained by a modified Hough conversion of straight lines; 

Figure 17 is an explanatory viev shoving arranged 
elements obtained by modified Hough conversion of straight lines 
and an example of a number of edge points located on straight lines 
corresponding to the respective arranged elements; 

Figure 18 is a table shoving straight lines of a large 
number of edge points; 

Figure 19 is an explanatory viev shoving contour segments 
vithin a small square; 



Figure 20 is a table shoving groaping by an edge 
orientation; 

Figure 21 is a table shoving grouping by a continuity; 

Figure 22 is an explanatory viev showing edge points on a 
small square for explaining grouping by a continuity; 

Figure 23 is a table shoving an example of a result of a 
perceptual grouping; 

Figure 24 is an explanatory viev shoving a pyramid guad 
tree structure; 

Figure 25 is a partial flov chart shoving details of the 

fourth step; 

Figures 26(a) to (d) are explanatory views shoving small 
squares to be searched next in the case of connecting contour 
segments; 

Figures 27 (a} to (d) are explanatory views shoving a 
method o£ determining an insertion direction from the form of a 
fold; 

Figure 28 is a partial flow chart showing a step of 
determining the centre of gravity of a ring; 

Figure 29 Is an explanatory view shoving an ellipse of a 

fold; 

Figure 30 is an explanatory viev shoving a step of 
determining an x-coordlnate of the centre of gravity of a ring; and 

Figure 31 is an explanatory view showing a step of 
determining a y-coordinate of the centre of gravity of a rlng- 

Figures 32 to 34 relate to the second embodiment of the 
present invention. 
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Figure 32 is an explanatory view shoving a part of an 
overlapped pyramid strnctore. 

Figoxe 33 is an explanatory view shoving a connection of 
contour segments in an overlapped pyramid stracture« 

Figure 34 is a flov chart shoving the foarth step of the 
embodiment. 

First of allj a somaiary of the present invention will be 
given by way of explanation vith reference to Figures 2 to 7. 

As sbovn in Figure 2, an endoscope 1 of the fibre-scope 
type is provided vith an elongated flexible insertion sheath 2 
connected to extend from the rear of a thick operating part 3. A 
flexible optical cable 4 extends from a side-port of the operating 
part 3 and is provided at its end vith a connector 5, tiAicb can be 
connected to a light output port of an illuminating source S 
containing a lamp 6d. The operating part 3 is provided at its rear 
end vith an eye-piece 8* 

As shovn in Figure 3, a rigid tip U is supported on a 
curvable section 12 provided at the remote end of the insertion 
sheath 2. The operating part 3 is provided vith a curvature 
control operating knob (not illustrated) so that the curvable 
section 12 may be curved vertically and/or horizontally, as 
required, by operation of this curvature control operating knob. 

An illuminating lens 15 of an illuminating optical system 
and an objective lens 16 of an observation optical system are 
arranged in the sheath end-face directed substantially in the same 
direction as the axis of the rigid tip 11« A light guide (not 
illustrated), for example an optical fibre bundle, is provided in 
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tbe sbeath 2 at the rear of the lllaminatiag lens 15. This light 
guide is inserted throagh the insertion sheath 2, operating part 3 
and optical cable 4, and is connected to the connector 5 so that, 
when this connector 5 Is coupled to the light source 6, the 
illuminating light emitted by the lainp 6a enters the light guide, 
to be led to the tip 11 and emitted from the tip end-face to be 
radiated out to an object through the illuminating lens 15, as 
indicated in Figure 3 by solid lines 17 representing the 
illuminated region. 

The end-face of an image-relaying fibre bundle (hot 
illustrated) is arranged in the image-forming plane of the 
objective lens IS. This image guide is inserted through the 
Insertion sheath 2 and extends to the eye-piece 8. The object 
image formed by the objective lens 16 will be led to the 
eye-piece 8 so that it can be observed through the eye-piece lens 
(not shown), which Is mounted in the eye-piece 8. In Figure 3, a 
pair of broken lines IB indicate the visual field of the observing 
optical system. 

Kow, as shown in Figure 3, the illuminating optical 
system and observing optical system of the endoscope 1 are adjacent 
to each other and are directed substantially in the same direction. 

On the other hand, annular folds, also. called haustras, 
exist on the inside wall of tbe large intestine » Most endoscope 
doctors judge the endoscope insertion direction by the manner in 
which such an annular fold is seen. That is to say, the centre of 
the ring of such fold is an excellent criterion in judging the 
endoscope insertion direction. This shall be explained with 
reference to Figures 4 to 7. 
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In Figures 5 and 7, folds 21 and 22 are represented on 
the inside vail of the large intestine. 

Figure 4 shows the case vtien inserting the insertion 
sheath 2 of the endoscope 1 into an upwardly bent part of the large 
intestine. In such a case, as shown in Figure 5, the folds are 
seen as being upwardly deviated* Therefore, in this case, the 
tip 11 of the endoscope 1 should be curved upward and the insertion 
sheath 2 will then be inserted to move upwardly. 

Figure 6 shows the case of inserting the insertion 
sheath 2 of the endoscope 1 into a straight part of the large 
Intestine 20. In this case, as shown in Figure 7, the folds are 
seen with no deviation, vertically or horizontally. Therefore, in 
this case, the Insertion sheath 2 of the endoscope 1 may be 
inserted in its straight condition. 

Thus, in the proposed methods of endoscope insertion 
direction detection, the form of the folds existing on the inside 
wall, as seen In the endoscope image, is e^rtracted and the 
endoscope insertion direction is deduced on the basis of the form 
of these folds. 

This embodiment of an endoscope insertion direction 
detecting method can be employed with endoscopes such as are shown, 
for example, in Figure 8 or Figure 9. 

The endoscope apparatus shown in Figure 8 is provided 
trith a fibre-scope 1 fed with illominating light by a light 
source 6, and an externally fitted television camera 30 is fitted 
to the eye-piece 8 of this fibre-scope 1. The construction of the 
fibre -scope 1 is the same as that shown in Figure 2 and need not be 
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further explained here. The externally fitted television camera 30 
is provided, for example, with an image lens (not illustrated) that 
forms an image of light from the eye-piece 8 and utilises a solid 
state imaging device (not Illustrated), arranged in the image plane 
of this image-forming lens. This externally fitted television 
camera 30 drives the solid state imaging device aod is connected to 
a signal processing apparatus 31 to convert the output signal of 
this solid state imaging device to form a video signal. The video 
signal output of the signal processing apparatus 31 is fed into a 
monitor 32, and also converted to digital form by an 
A/D-converter 33, which then feeds into an electronic computer 35, 
and can be taken into a memory (not illustrated} vl thin this 
electronic computer 35. The endoscope image will be displayed on 
the monitor 32, and the endoscope Insertion direction detecting 
method in this embodiment can be carried out by the electronic 
computer 35. 

The endoscope apparatus shown in Figure 9 is provided 
with a video-scope 41 instead of a fibre-scope 1 and externally 
fitted television camera 30. As with the fibre-scope 1, this 
video-scope 41 is provided with an elongate flexible insertion 
sheath 2 and an operating part 3 connected to the rear of this 
insertion sheath 2. A flexible optical cable 42 extends sldewise 
from the operating part 3 and is provided at its end with a 
connector 43 yhlch is connected to a control apparatus 45 
containing a light source 6 and a video signal processing 
circuit 46. A solid state imaging device (not illustrated) is 
arranged in the image forming position of the objective lens in the 
tip of the insertion sheath 2 of the video-scope 41 and is 




10 

connected to a video signal processing circuit 46 vithin the 
control apparatus 45 through the signal lines Inserted through the 
insertion sheath 2, operating part 3^ cable 42 and connector 43. 
The Illuminating optical system of the video-scope 41 is the same 
as that of the fibre-scope 1, in that the i nominating light 
emitted from the lamp 6a of the light source 6 vithin the control 
apparatus 45 may enter the light guide at the entrance end. The 
solid state imaging device will be driven by the video signal 
processing circuit 46, and the output signal of this solid state 
imaging device will be processed to form a video signal by the 
video signal processing circuit. As in the endoscope apparatus 
using the fibre-scope 1, the video signal output from this signal 
processing circuit 46 can be fed into the monitor 32, and can be 
converted to digital form by the A/D-converter 33^ to be input into 
the electronic computer 35 and taken into the memory (not 
illustrated) vithin this electronic computer 35. The endoscope 
image vill be displayed on the monitor 32 and the endoscope 
insertion direction detecting method in this embodiment will be 
carried out by the electronic computer 35. 

The endoscope insertion direction detecting method 
utilised in this embodiment vill be further explained in the 
folloving, vitb reference to Figure 1. 

As indicated in the flow chart shovn in Figure 1, this 
direction detecting method comprises a first step 51, extracting 
discontinuous points in an original image fed into the electronic 
computer 35, a second step 32 dividing the picture image obtained 
in the first step SI into a plurality of picture images and 
extracting contour segment candidates by using a Mdif ied Hough 
conversion from each of the divided picture images, a third step S3 
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making perceptoal grouping of the contour segment candidates 
obtained in the second step and extracting the optimum contour 
segment from each of tbe divided pictnre images, a fourth step S4 
connecting the contour segments obtained in the third step S3, and 
a fifth step 35 for deterMning the ineertion direction from the 
result of the 
fourth step 54. 

The first step will be explained %rith reference to 
Figures 10, 11a and lib. 

In extracting discontinuous points, a colour defined by 
red, green and blue values may be noted, or a grey level (thickness 
or brightness) • In this embodiment, the case o£ noting a grey, 
level will be explained.. The number of pixels of the original 
picture shall be 512 x 512 and the grey level shall be of 256 
gradations. 

Extracting discontinaous paints by noting a grey level is 
achieved by inspecting the variation rate (gradient) of the grey 
level on a spatial coordinate, and extracting the grey level In the 
variation. This is an edge detection by noting the grey level. 

For the edge detecting, for exasvle, spatial filterlTig 
using a overlapped matrix Is used In this embodiment. 

Spatial filtering for the case using an overlapped 
matrix consisting of 3 x 3 pixels vill be explained with 
reference to Figure 10. In Figure 10, Pi (xi,yi) 
represents the grey level of the pixel of an Input picture 
Image Pi having a coordinate (xi,yi]. In the same 
manner, P2 (xi,yi) represents tbe gradient of the picture 
image of coordinate (xi,yi) in an output picture image P2. 
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First of all, the locality of 3 x 3 in the input picture 
image Pi (xi,yi) is taken out, a product of the value of each 
pixel in this local zone of 3 x 3 calculated, and the value of each 
element corzespondlng to the overlapped matrix consisting of 
separately prepared 3x3 elerants is calculated, to determine a 
sum of 9 products for the zone P2 (xi^yi) in the output 
picture image. 

This operation is repeated in turn for the respective 
pixels of the input picture image to obtain an output picture 
image P2 to %rtiich spatial filtering has been applied* 

Nov, a gradient (grey level variation rate) gx in 
the x-direction is obtained by using an overlapped matrix as shown 
in Figure 11(a). In the same manner, a gradient (grey level 
variation rate] gy in the y-direction is obtained by using an 
overlapped matrix as shown in Figure 11(b). An absolute value of 
the gradient in a pixel is given by the under-=-mentioned 
formula (1-1) but, in this embodiment, the absolute value need not 
be very accurate and therefore, for brevity in the operation. It 
may be approximated by a formula {1-2} as follows:- 



g a gx2 + gy2 



(1-1 )• 



g = Igxl + I9yl 
where g represents the magnitude of the discontinuity. 



(1-2). 
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For the pixel of the coordinate (xi^yi) in the case 
of the overlapped matrices shovn in Figures ll(a} and ll(b)« 9x 
and gy are used, and are given In concrete terns by the folloving 
formulae, (1-3) and (1-4):- 
gx = P2 (Xi,yi) 

« -Pi (xj - 1/ yi + U 
♦Pi (xi + 1/ yi + 1) 
'■fT • Pi (Xi - 1, yi) 
^-fT • Pi (Xi + 1 + yi) 
-Pi (xi - 1, yi - 1) 

+Pi (xi + 1 + yi - 1) (1-3). 

gy « P2 (xi/Yl) 

= Pi (XI - 1/ Yi + l) 
t-f?". Pi (XI + yi + 1) 
+Pl (XI +. 1, yi + 1) 
-Pi (XI - 1 ♦ yi - 1) 
-fT . Pi (XI, yi - 1) 

-Pi (Xi + 1 + yi - 1) (1-4). 
The direction of the edge is given by the following 
formula (1-5):- 

6 = arctan (gy/gx) (1-5). 
Here, g is determined by the formula (1-1) or (1-2), and 
is cooqpared vitb a predetermined reference value gx, and edge 
points above the reference value gx are not kept. 
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To redace the nomber of discarded edge points one coald 
set the reference value 9r to be a large value, but If the 
reference value gr Is set to be too large a value, then 
Inherently necessary edge points are likely to be discarded, and 
therefore it is ioportant to set the value gr to be rather lov. 
It is preferable to set the reference value gr so as to leave 
about 80% of all tho edge points. It oay be set so that about 60% 
of the edge points zeioain after the dispersion of g of all the edge 
points is determined. 

Thus, when the somewhat small value of g^ is used and 
the perceptual grouping is then made, a veak but significant edge 
point can be extracted without being influenced by noise or the 
like. This is one of the advantageous features of this embodiment. 

The second step will nov be explained with reference to 
Figures 12 to 19. 

First of all, as shorn in Figure 12, the output picture 
image P2 (xi,yi) obtained in the first step is divided 
into small squares of about 8x8 pixels and is converted in a 
modified Bough manner. Where an original picture image consists 
of 512 X 512 pixels, it vill !ie divided into 64 x 64 picture 
images. Thus, in this embodiment. It is divided into small squares 
of 8 X 8 pixels by way of example/ but, depending on the required 
precision, it may be divided Into 4x4 pixels, or other values as 
desired. 
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An output picture Image P2 (xi/yi) and a selected . 
si&all square o£ 8 x 8 pixels are shown respectively In Figures 12 
and 13. As shown in Figure 14^ if a small square o£ from 4x4 
to 16 X 16 pixels is selected, through which the fold passes, the 
short length of fold within the si&all square can be considered as a 
substantially straight line* 

First of all, we should briefly explain the modified 
Hough conversion* 

Figure 13 is a simplified representation of a coismon, 
well used x,y coordinate system. In order to make a modified Hough 
conversion, then as shown in Figure 15, addresses are attached all 
round the outer periphery of the small square of 8 x 8 pixels. 

Then, the straight line on the small square of 8 x 8 
pixels can be defined by designating a start address (S) and end 
address (E). For exai^le, the straight line represented by a, 
broken line in Figure 15 can be defined as a straight line having a 
start address 3 and an end address 20 « 

On the other hand, the kinds of straight lines which 
can be described on the small square of 8 x 8 pixels 
are (32 x 12]/2 » 512. The reason for multiplying (32 x 32) 
by 1/2 is that the straight line on the small square is not a 
vector. That is to say, the straight line between the start 
address 3 and end address 20 can be considered to be identical to 
the straight line between the start address 20 and end address 3. 

Therefore, all the straight lines correspond to one of 
the elements as arranged in Figure 16. 
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Vhen any edge points on the straight line corresponding 
to one arrangement on the arranged eleir»nts are present within any 
one of the arranged elements, for exas^le, considering the straight 
line shovn as a broken line in Figure 15, a straight line having a 
start address 3 and an end address 20, it passes tfarongh four small 
squares having four edge points, and the line vill therefore be 
expressed as in Figure 16. 

If a modified Hough transformation in accordance with 
such an idea, in fact, a result such as is shown in Figure 17 will 
be 

obtained. As already described, one of the arranged elements 
(one of the squares in Figure 17} corresponds to one straight line 
and the numeral (vote) within the arranged element represents the 
number of edge points existing on that straight line. 

The larger the number of edge points existing on any 
straight line, the higher is the possibility that this is in fact 
the required straight line. 

Therefore, about 5 votes are extracted as contour segment 
candidates from the larger number of votes. Such extracted contour 
candidates bears a markO^^ Figure 17 and are shown as a table in 
Figure 18. 

Bere lies an Idea of extracting the largest number of 
votes and making it a required contour segment. However, if it is 
made so, a wrong contour segment is likely to be extracted, for 
example, as in the case shown in Figure 19 in which (a) is a 
contour segment to be extracted, whereas, if judged only by the 
number of votes, the contour segment (b) will be erroneously 
extracted because of noise content. 
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In order to avoid such danger^ the number of votes is 
reduced to about 5 arranged eleioents, any constant grouping is then 
identified, and the optimum contour segment can then be extracted. 

The perceptual grouping of the third step vill nov be 
explained vith reference to Figures 20 to 23. 

In this embodiment, tlie perceptual grouping Is made by 
noting the three items defined below* However, in addition to the 
under -mentioned three Items, the size of the gradient (edge 
magnitude) may be noted, or the colour may be noted. The items 
arex- 

1. Edge orientation 
(on edge points] ; 
Reference: ± 22.50 

2. Grey level 
(on pixels) : 
Reference; dl 4 level 

3. Continuity 

(on edge points}: 
Reference; ±1 pixel distance 
Bach of the above-mentioned three items will nov be 

explained* 

For example. In the case of Figare 17, the following 
processes are made on each of the five lines in Figure 17:- 
1, Mae orientatlon;- 

For ejQimple, the line 4, the arranged element (16,4) in 
Figure 18 has 9 edge points arranged in the order of the smaller 
edge direction 0 determined in the first step, as shown in 
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Figure 20. A new gzoaplng Is made %4iere there is a gap larger 
than 450. In the examples shovn in Figure 20, the difference 
of 9 between the 8th edge point and 9th edge point is : 

620 - 350 = 270 (> 22.50).. 
Here, a regrouping is made. 

As in the case of the above-isentioned edge orientation, 
the grey level o£ the parts corresponding to the edge points are 
arranged in the order of the smaller one and a grouping is made 
where the gap of the grey levels is larger than 4* 

3. Contlnnitv!- 

The x-coordlnates and y-coordinatesof the edge points are 
noted and are arranged in the order of the soaller x-coordinate as 
shorn in Figure 21. The same x--coordinates are arranged in the 
order of the larger ^coordinate. Where the difference of the 
x-coordinate is ^x > 1 or the difference A/ of the y-coordinate 
is Ay > 1, a grouping is made. In the example shown in Figure 21, 
the difference of the y-coardinate between the 4th edge point 
and 5th edge point is 2 and here a grouping is made. Thus, by 
noting both of the x-coordinate and y-coordinate, even if the 
straight line rises extreraely or lies level with respect to the 
X-axis or y-axis, the discontinuous point can be positively 
extracted and a grouping can be made. For example/ in Figure 22, 
the respective edge points in the example In Figure 21 are plotted 
on the x and y-coordinates. In such a case, if only the 
x-coordinate is noted, it Is not possible to detect that the 
straight line is discontinuous. 
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By inaking the above three operations, generally such 
results, as are shovn for example In Figure 23, are obtained. 

Here, Eor example, if a logical formula of 

Continuity n (Grey Level )n (Mge Orientation} (1) 
is applied as a condition of extracting the optimum contour 
segment, as shown in Figure 23, the edge points can be grouped in 
six groups of A to F. In the example in Figure 23, the group A has 
the most edge points and the number of the edge points is 6. 

The condition of extracting the optimum contour segment 

is 

not limited to the logical formula (1} but the following logical 
formula (2} may be used, for example:- 

Continuity O (Grey LevelL/ Edge Orientation) (2). 

The same is carried out also on the other arranged 
elements and the group having the most edge points among them is 
extracted and forms a contour segment to be extracted in the small 
square of 8 x 8 pixels. Thus, in the third step, a contour segment 
of 64 x 64 small regions consisting of 8 x 8 pixels could be 
extracted. (Needless to s^y, there are many small regions In lAich 
no contour segment exists}. The division into small regions gives 
advantages, in that parallel processes by a plurality of coi^oters 
are possible and the operating time can be reduced. Parallel 
processes may be made by using a plurality of exclusive IC's. 

Furthermore, also in the case of noting the size of the 
gradient (edge magnitude) or the colour, in the same manner, the 
edge points may be arranged in the order of the size of the 
gradient or the colour and may be grouped where there is a gap 
larger than that predetermined. 
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The fourth step will nov be explained vlth reference to 
Figures 24 to 26. 

In this fourth step, the contoor segments obtained In the 
third step are connected. This is called a tracing or connecting 
o£ edges. 

In tracing the edges, that segnsnt from which the tracing 
Is to be started Is important. In this embodiment, a pyramid quad 
tree structure Is utilised to trace the edges, k step of obtaining 
a curve of a fold by utilising this pyramid quad tree structure 
will now be explained with reference to Figures 24 and 25. 

First of all/ as shown in Figure 24, a pyramid quad tree 
structure is made by making a small region, consisting of 8 x 8 
pixels, a leaf node (or leaf). That is to say, in the step 541 
shown in Figure 25, a small region of 8 x 8 pixels is made a leaf 
node and, in the step 542, the sum of the contour segment numbers 
of four children nodes is made a contour segment number of a parent 
node to obtain a picture image of n/2 x n/2 pixels. Then, through 
the step S43 of Judging whether the number of pixels is 1 or not, 
the step S42 is repeated until a root (or root node) is reached. 
In the thus made pyramid quad tree structure, the parent node 
retains the number of the contour segments of the children nodes. 
In Figure 24 the figures near the respective nodes represent the 
numbers of contoor segments. 

Then, in order to discover a starting contour segment, 
the tree is searched downward from the root, that is to say, in the 
step 544, the operation o£ selecting that child node having the 
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most contour segments aiiK>ng the four son modes Is repeated. As 
shorn in Figure 24, in the stage of the leaf, in a case where a 
plurality of leaves having contour segments exist, any contour 
segment may be xoade a starting contour segment. 

Then, In the step S45, the contour segment determined in 
the step S44 is made a starting contour segment and the contour 
segments are connected. 

The operation of connecting these contour segments will 
now be explained with reference to Figures 26(a} to (d). 

As shown in Figure 26(a), in case the contour segments 
are connected in the order of the snail regions 1 and 2, the next 
searching direction will be the direction indicated by the arrow in 
the drawing. Therefore, In this case, the small regions a, b and c 
are checked. 

In the same manner, as shown in Figure 26 (b), in case the 
contour segments are connected in the order of the small regions 1 
and 2, the next searching direction will be the direction indicated 
by the arrow in the drawing. In this case, the small regions a, b 
and c are checked. 

As shown in Figure 26 (c), in case the contour segments 
are connected in the order of the small regions 1 and 2, the small 
regions a, b and c are checked but, in case the contour segments 
exist in both of the small regions a and c, the directions of the 
contour segments a and c are checked and the direction in which a 
smooth connection is made is selected. In the case of Figure 
26 [c), the contour segment of the small region c is selected. 
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As shown in Figure 2€(d), in case the contour segments 
are connected in the order of the siaall regions 1 and 2 and no 
contour segment exists in the small regions a^ b and c, the small 
regions d, e, £, g, b, i and J are inspected, because, in case the 
contour segment a is considered to have lapsed for any reason, the 
next search tflll be of d, e and f, in the sbjdb manner, in case the 
contour segment b is considered to have lapsed, the next search 
vill be of f , g and h and, in case the contour segment c is 
considered to have lapsed, the next search .vill be of h, i and i. 

In case the contour segments are to be connected, the 
angle formed by tvo contour segments to be connected may be limited 
to be vithin±4S0. 

Whenever the contour segments are connected, the 
numerical value on the quad tree vill be corrected « That is to 
say, the 

value of the leaf corresponding to the connected contour segment 
vill be changed from 1 to 0. The value of the parent node above it 
vill also be corrected. 

The contour segments are connected in both directions « 

ghen a curve is thus obtained, in the step S46, vhether 
the fourth step is to be concluded or not is judged and, in case It 
is not be be concluded, vhen the steps in and after the step S41 
are again repeated, another curve can be obtained. 

The fifth step shall be explained in the folloving vith 
reference to Figures 27 to 31« 

In the fifth step, the insertion direction is judged and 
determined by the form of the fold determined in the fourth step. 
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The form of the fold to be determined is of such pattern 
as is shown, for eaeaople, in Figures 27(d} to (d). The insertion 
direction is determined respectively by the methods explained in 
the following. 

Figure 27(a) shows the case that two carves (folds) G, 
Ci and C29 having an intersection (or branching point) 
indicated by the mark * in the drawing are obtained. In this case, 
by judging the direction of the contour segment at the point 
indicated by the mark * in the drawing, the inner fold is Judged. 
In the 

case of Figure 27(a), the curve C2 is inner and the centre 
(point a] of the inner curve is made an insertion direction. In a 
case where such a curve is obtained, it is considered to be a case 
tAere such a tube cavity of the large intestine in which an 
endoscope is inserted is curved, and the inner fold is partly 
hidden. 

In the method shown in Figure 27(b), radial lines are 
constructed respectively at about five points on the obtained curve 
and the part in which the intersections of the radial lines 
concentrate is made the insertion direction. 

In the method shown in Figure 27(d), in any case where 
the obtained curve is broken anywhere, the nearest curves are 
connected and the centre of gravity of the obtained ring is made an 
insertion direction. 

In the methods shown in Figures 27(b) to (d), any curves 
or rings may be used, the largest curve or ring may be used or the 
ring n-th from a large ring may be predetermined to be used. 




24 

The centre of gravity can be determined by such a process 
as is shown, for example, in Figure 28 « 

First of all, in the step S51, the number of pixels 
contained in the circle or ellipse of a fold is determined and is 
made R as shovn in Figure 29. The nomber of pixels may be replaced 
with the namber of small squares used in the step. 

Then, in the step S52, as indicated by the arrows in 
FIgare 30, the nomber of pixels contained in the circle or ellipse 
of the fold is cotxnted in the direction of the x-axis from above 
until the namber becomes N/2. The value of the y-coordinate when 
the namber becomes ir/2 is represented by yo which is a 
y**coordinate of the determined centre of gravity. 

In the same manner, in the step S53, as indicated by the 
arrows in Figure 31^ the number of pixels contained in the circle 
or ellipse of a fold is counted in the direction of the y-axls from 
the left side until the number K/2» The value of the x-coordinate 
when the namber becomes N/2 is represented by xq which is an 
x-coordinate of the determined centre of gravity. 

Then, in the step the centre of gravity of the fold 

is determined as (^OiYo)- form of the fold has been 

described as circular or elliptical, but is hot limited to such 
shapes . 

As the endoscope is inserted, the form of the fold will 
vary. Therefore^ the centre of gravity of the fold n-th as counted 
from a large one may always be determined and may be Judged to be 
an insertion direction or an average value of the centre of gravity 
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of the n-th fold as counted from a large one and the centre of 
gravity of the n -f m-th fold nay be determined and jadged to be an 
Insertion direction. Also, the centre of gravity of the fold on 
the innermost side may be made an Insertion direction. The 
direction in vhich the most centres of gravity among the centres of 
gravity of a plurality of folds are located may be made an 
insertion direction. 

As shown in Figure 4, in a case where the large 
intestine 20 is curved, then as shorn in Figure S, the centre of 
gravity of the fold is different depending on the fold. In the 
case of Figure 5, the more the centre of gravity is on the Inside 
of the fold, the more the insertion direction side is on the upper 
side. The larger the curvature, the larger the displacement of 
this centre of gravity. Therefore, the curvature of the large 
intestine 20 can be detected from the displacements of the centre 
of gravity of a plurality of folds. 

Thus, according to this embodiment, the form of a fold is 
determined by the first to fourth steps, the endoscope insertion 
direction is judged in the fifth step on the basis of the form of 
the fold and thereby the endoscope insertion direction can be 
simply detected. 

In the case of extracting a discontinuous point 
(edge point) in the endoscope picture by the first step, iiAen the 
reference value is set to be rather low and the value of the 
gradient g Is left to be small to some extent, a necessary 
discontinuous point (edge point) will be able to be extracted 
without being influenced by noise or the like. 
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In cases where contour segment candidates are extracted 
by utilising a modified Hoogh conversion in the second step or in 
cases where the optimura contour segment is extracted from the 
respective divided plctore Images by a perceptual grouping in the 
second step, then by dividing the endoscope picture image into 
small regions, parallel processing by a plurality of computers is 
made possible and the operating time can be reduced. 

As a pyramid gnad tree structure is utilised in 
extracting a starting contour segment in the fourth step^ the 
processing time can be remarkably reduced. 

Incidentally, in this embodiment, a grey level is noted 
In extracting discontinuous points in the first step, but a colour 
may equally be noted, as described above. 

In case a colour is noted, for example, the variation 
rates of- the hue and saturation may be inspected and the hoe and 
saturation as varying may be extracted. 

For exan^le, in case three primary colour components 
(three exciting values) R, 6 and B of a CIE-SGB colouring system 
are obtained from an original picture, a hue 0 can be represented 
by a formula (2-2), by using the following formula (2-1):- 
^ 2r g - b 

©i = cos-1 (2-1) 

(6 f(r-l/3)2 f (g-l/3)2 + (b-l/3)2) 

^ ^^1 (g^ h), 2v6i (g < b) (2-2) 

where: r = R/(R + G + B); 

g <^ 6/(R t G -I- B); and 
b = B/(R ♦ G ♦ B). 
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The saturation S can be represented by the following 
fomuila (2-3):- 

S « 1 - 3 min (r, b) (2-3) 
where min (T/ q, b] represents the minioum value of r, g and b« 

Thus, when the hue and saturation are made numerical 
values for respective pixels of the original picture, then lust as 
in the case of noting the grey level, by a spatial filtering or the 
like, the hue and saturation variations can be extracted. Just as 
in the case of noting a grey level, by carrying out the second to 
fifth steps, the fold can be extracted by noting the colour. 

In a case where the original picture is given by an NTSC 
signal, the hue can be obtained from the phase of a chrominance 
signal and the saturation can be obtained from the amplitude of the 
chrominance signal. 

Also, the value of a specific colour component may be 

noted . 

In Figures 32 to 34, the second embodiment of the present 
Invention is shown. 

In this embodiment, as cos^iared with the first 
endbodiment, the fourth step, that is the step of connecting contour 
segments is different* 

In this embodiment, a plurality of contour segments are 
simultaneously connected by using an overlapped pyramid. 

The overlapped pyramid structure used in this embodiment 
is shown in Figure 32. In this overlapped pyramid, respective 
nodes have 16 children nodes and 4 parent nodes. In Figure 32, 
4 X 4 (»16) nodes enclosed with fine solid state lines belong to 
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one parent node. That Is to say, children nodes corresponding to a 
plurality of parent nodes are overlapped. ThuS/ one child node has 
4 parent nodes. Soch an overlapped pyramid strnctnre is isade by 
considering a small region consisting of fl x 8 pixels as a leaf 
node. 

Then, by nsing this overlapped pyramid structure, a 
method of connecting contour segments will be explained. 

In Figure 33, one parent node and 16 children nodes 
corresponding to the parent node in Figure 32 are parted and sho%m. 

First of all, in case there are a plurality of line 
segments in a region consisting of 2 x 2 (^4) nodes enclosed with 
thick solid lines among IS children nodes In Figures 32 and 33, it 
will be examined whether these segments can be connected. Here, 
the directions and edge points of two contour segments (which are 
hereinafter called connecting Information) are considered to 
determine whether the two contour segments are connected. For 
example, if the difference of the direction between two contour 
segments Is ASo or less, and also the distance between the edge 
points Is a predetermined distance threshold J^smx or less, two 
contour segments are considered to be connected. 

In Figure 33, two sets of contour segments connected in 
accordance with the above mentioned rule are shown. In the exanyile 
shown in this Figure, there are contour segments (a), (b), (c) and 
(d) within the respective 2x2 nodes. Since the difference in 
direction between contour segments (a] and (b) is 450 or 

less and also the distance Di between edge points is a distance 
threshold Dioax or less, they are considered to be connected as 
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the same group. Siinilarly/ since the differeoce in direction 
^2 between the two contour segments (c) and (d] is 450 or 
less, and also the distance D2 betveen edge points is a distance 
threshold Dwrx or less, they are considered to be connected as 
the same group. 

The coordinates (Y3, Y4, Y5 and Ye in Figure 33) 
of the edge points at which the two sets of lines connected in this 
manner meet the perioeter enclosing 4 children nodes (shown by 
thick solid lines}, and the angles I & 3^ ^ 4^ ^5 and 
®S in Figure 33) at which the two sets of lines intersect the 
perimeter, are raised to a parent node as connecting information of 
the parent node. 

Then, a plane to which the above mentioned parent node 
belongs is considered. That is, 4 parent nodes as shown In 
Figure 32 have the connecting information raised from their 
children nodes, respectively. In the regions consisting of these 4 
parent nodes, contour segments are connected in accordance with the 
above mentioned rule. 

The same is repeated up to the root level and the whole 
connection of contour segments is completed. 

The above mentioned operation will be explained by using 
a flow chart in Figure 34. 

First of all, a small region consisting of 8 x 8 pixels 
is made a leaf node in step S61. 

Then, in step S62, the coordinates of the points at ^ich 
contour segments In respective nodes meet the perimeters of the 
nodes. 
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and the angles at lAich the contour segments and the perineters 
intersect, are amde connecting Information. 

Hext in step SS3, when there are a plurality of contour 
segments in a region of 2 x 2 nodes and vben the connecting 
information satisfies predetermined conditions, that is to say, 
when the difference in direction between two contoar segments is 
450 or less, and also the distance betveen edge points is a 
predetermined distance threshold Dnex or less, contour segments 
are connected. 

Next, in step 8S4, a plane consisting of parent nodes 
having 4x4 overlapped children nodes is formed, and the 
coordinates of the points at which contour segments connected 
betveen 2x2 children nodes of the central part among 4x4 
children nodes meet perimeters enclosing 2x2 children nodes 
(corresponding to a 

perimeter of a parent node}, and the angles at which the contour 
segments and the perimeters intersect, are made the connecting 
information In the parent node« 

Then in step S65, it is judged whether the number of 
nodes is one or not< If not, the process goes back to the above 
mentioned step BS3 and steps S64 and S65 are carried out in the 
level of parent node. If the number of nodes is one, the process 
is completed. 

Thus, the formation of the overlapped pyramid and the 
connection of contour segments are carried out in parallel. The 
formation starts from a leaf level and when the root level is 
reached, the whole contour segments are connected. 
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The fourth step of the f ixst erabodlj&ent is a method 
Incladlng the folloving steps. A fold is extracted by connecting 
contour segments one by one and the reioalning contour segments are 
connected one by one in order to extract the next fold, fiovever^ 
in the fourth step of this embodiment, a plurality of folds are 
extracted In parallel so that processing time can be shortened. 

Uso, according to the method of this embodiment, since a 
pyramid Is overlapped and formed, the contour segments on a 
boundary of a region are not overlooked. 

Other information, operation and effects are the same as 
the first embodiment. 

The endoscope operator may insert the endoscope by the 
curving operation in the endoscope insertion direction detected by 
the method of the present Invention, or the endoscope may be 
inserted by automatically directing the tip part by using the 
apparatus in the manner described. 

As explained above, according to the present invention, 
by extracting the form of a fold and using a judgement on the basis 
of this form of the fold, the endoscope, insertion direction can be 
simply detected. 

Also, by dividing an endoscope picture image into small 
regions in extracting contour segments and by utilising a pyramid 
quad tree structure in connecting the contour segments, there is an 
effect of reducing the processing time. 
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In the case of extracting dlscontlnoous points In an 
endoscope picture iinage^ by setting the reference value to be 
rather lov, extracting contour segaient candidates and then 
extracting the optimum contour segment by a perceptual grouping, 
there are the effects that the influence of noise or the like can 
be reduced, the form of the fold can be extracted more accurately 
and the insertion direction can be detected. 

It is apparent that, in this invention, implementation 
methods differing over a vide range can he formed without departing 
from the spirit and scope of the invention. This invention is not 
restricted by its specific working modes except being limited by 
the appended Claims. 
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PATEMT CLAIM5; 

1. A method of detecting a required endoscope insertion 
direction inclnding the step of extracting the form of a fold 
existing on the inside wall of an observed part from an endoscope 
picture image so that the endoscope insertion direction may be 
judged on the basis of the form of the fold extracted by said step. 

2. A method according to Claim 1, wherein said fold form 
extracting step is to detect Ja part at tdiich the brightness or 
colour of the picture iiaage varies and to consider said part to be 
a fold. 

3. A method o£ detecting a reanired endoscope insertion 
direction comprising the respective steps of: 

extracting discontinnons points in an endoscope picture 

image; 

extracting contour segments based on said discontinnous 
points from respective divided picture images obtained by dividing 
the picture image obtained by said discontinuous point extracting 
step into a plurality of picture images; 

connecting the contour segments obtained by extracting 
said contour segments; and 

considering the line connected by said connecting step to 
be the form of a fold existing on the inside wall of an observed 
part and judging the endoscope insertion direction on the basis of 
said form of the fold. 



- • 



34 

4. A method according to Claim 3, wherein said discontinuous 
point extracting step includes extracting points at which the 
brightness of the picture image varies. 

5. A method according to Clalxn 3, wherein said step of 
extracting discontinuous points includes extracting points at which 
the colour of the picture Image varies. 

A method according to Claim 3, wherein said step of 
extracting discontinuous points includes extracting discontinuous 
points by applying a spatial filtering by using an overlapped 
matrix » 

7. A method according to Claim 3, wherein said step of 
extracting contour segments has a step of extracting contour 
segment candidates and a step of extracting the optimum contour 
segment from among the candidates extracted by said candidate 
extracting step. 

8. A method according to Claim 7, wherein said candidate 
extracting step includes using a modified Hough conversion in 
extracting the contour segments. 

9. A method according to Claim 7, wherein said modified 
Hough conversion includes defining the contour segments vttbin the 
divided picture image by addresses attached to the periphery of the 
divided picture image. 
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10. A method according to Claim 7, wherein said optimum 
contoar segment extracting step includes carrying out a perceptual 
grouping to extract the optimum contour segment. 

11. A method according to Claim 3, vherein said optimum 
contour segment extracting step utilises at least an edge 
orientation, grey level, continuity, edge magnitude or colour in 
carrying out said perceptual grouping. 

12. A method according to Claim 3, vherein said connecting 
step includes extracting a search starting contour segment by 
utilising a pyramid goad tree structure in connecting said contour 
segments » 

13. A method according to Claim 3, vherein said connecting 
step includes determining the next searched divided picture image 
in response to the arranging direction of the divided picture 
images having so far connected contour segments. 

14. A method according to Claim 3, «Aerein said connecting 
step includes limiting the angle formed by tvo connected contour 
segments to be vltbind^4So in connecting said contour segments. 
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15. A method according to Claim 3^ wherein, In a case irtiere 

tifo curves having an Intersection are obtained by said connecting 
step, said endoscope insertion direction judging step includes 
judging the direction of the contour segmnt at said intersection, 
thereby jodging which curve is inner and then judging the centre of 
the inner curve to be the insertion direction. 

IS. A Doethod according to Claim 3, wherein. In a case where a 

ring-like line is obtained by said connecting step, said endoscope 
insertion direction judging step includes judging the centre of 
gravity of said ring-like line to be the Insertion direction. 

17. A method according to Claim 3, herein said endoscope 
insertion direction judging step includes judging the part in which 
the intersections of radial lines erected respectively from a 
plurality of points on the curves obtained by said connecting step 
concentrate to be the required insertion direction. 

18. A isethod according to Claim 3, wherein, in a case where 
the curve obtained by said connecting step is like a partly incised 
ring, said endoscope insertion direction judging step includes 
connecting the obbalned curve at both ends and judging the centre 
of gravity of the thereby obtained ring to be the insertion 
direction. 
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19. & method according to Clalo 3, wherein, in a case irtiere 
the carve obtained by said connecting step Is partly broken, said 
endoscope insertion direction judging step inclndes connecting the 
nearest curves and judging the centre of gravity of the obtained 
ring to be the insertion direction. 

20. A method according to Claim 1 or Claim 3, wherein said 
endoscope picture image is obtained by a television camera fitted 
to the endoscope eye-piece capable of naked eye observation » 

21. A method according to Claim 1 or Claim 3, wherein said 
endoscope picture image is obtained by an imaging means provided in 
the endoscope. 

22. A method according to Claim 3, wherein said connecting 
step includes forming a pyramid of picture images making picture 
images consisting of an assemblage of said divided picture images a 
bottom layer; and 

connecting the contour segments in order from the bottom 
layer of the pyramid of these images to the root node. 

23* A method according to Claim 22, wherein a formation of 

said pyramid of picture Images and connection of contour segments 
are carried out in parallel In said connecting step. 

24. A method according to Claim 23, wherein said connecting 

step includes connecting two contour segments for which the 



coordinated of the point at lAlch tbe contour segments in 
respective nodes and perimeters of the nodes meet, and the angles 
at which the contour segments intersect the perimeters, satisfy 
predetermined conditions an^ng a plurality of children nodes 
belonging to a certain parent node in said pyramid of picture 
images. 

25. A method according to Claim 24, vherein said 

predetermined conditions are that the distance between said points 
is a predetermined distance or less and also the difference of said 
angles is a predetermined angle or less. 

2S* A method according to Claim 24, vhereln said pyramid of 

picture images is formed %Alle the coordinates of the points at 
which the contour segments connected among a plurality of children 
nodes and lines corresponding to a perimeter of a parent node meet^ 
and the angles at %rtiich the contour segments intersect perimeter 
lines, are made the connecting information of the parent node. 

27 » A method according to Claim 22, irtierein said pyramid of 

picture images has an overlapped pyramid structure in which 
respective children nodes corresponding to different parent aodes 
are overlapped. 
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